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(54) Infiage conversion and encoding techniques 

(57) A method of encoding a 2D video signal with 
conversion data to assist in the conversion of said 2D 
video signal to left and right eye images for a stereo- 
scopic display, characterised in that said method in- 
cludes: 

adding conversion data to said video signal to pro- 
vide the encoded signal, the data defining the displace- 
ment of respective selected points of said video image 
for converting the video image to a stretched image for 
a stereoscopic display. 
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Description 

[0001] The present invention Is generally directed to- 
wards stereoscopic Image synthesis and more partlcu- 
lariy toward a method of converting two dimensional 
(2D) images for further encoding, transmission and de- 
coding for the purpose of stereoscopic image display on 
two dimensional (2D) or three dimensional (3D) stereo- 
scopic displays systems. 

[0002] Recent improvements in technology in the ar- 
eas of compact high perfomiance video projection sys- 
tems, image processing, digital video and liquid crystal 
panels have made possible many practical 3D display 
systems utilising both active and passive polarising 
glasses and both single and multi-viewer autostereo- 
scopic displays. 

[0003] Three dimensional display,systems have 
moved out of the arena of being technological curiosities 
and are now becoming practical display systems for en- 
tertainment, commercial and scientific applications. 
There has now emerged the requirement for 3D media 
to display on these devices. Traditionally there has been 
just two ways to produce this 3D media (ie media pro- 
duced that contains image information for at least two 
separate views of the same scene from different per- 
spective's). These are: 

1 ) Generation of two separate views (usually in real 
time) by a computer. 

2) Videoing or filming with two laterally displaced 
cameras 

[0004] In the case of computer generated Images for 
usage in Computer Aided Design (CAD) systems, sim- 
ulators or video game equipment, il is not a complex 
process to produce two separate Images with different 
perspective's. 

[0005] The filming of movies utilising two laterally dis- 
placed cameras to produce 3D has been well under- 
stood for many years. However, there are many prob- 
lems with this approach. It Is considerably more difficult 
to film or video in 3D than in 2D because there are limits 
to the pemnissible distance between the nearest and far- 
thest objects in the scene (practical 3D depth of field) 
as well as framing problems (such as near objects being 
seen on only one camera and thus highlighting the in- 
accuracy of the 30 image generation when re-played. 
Another problem is maintaining a smooth pan without 
causing false 3D artefacts due to latency between the 
images from the two cameras and so on. 
[0006] Because of the complexity, high cost of pro- 
duction and Implementation, and the fact that there are 
as yet still only a very small number of 3D display sys- 
tems being produced for the domestic and commercial 
markets there has not been a large incentive for the ma- 
jor producers of films or videos to produce 3D media. 
IHowever, if a technique was devised that would allow 
conventional 2D films to be re-processed Into a 3D ver- 
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slon then it would be possible to not only convert new 
films Into 3D format for significantly less cost than filming 
them directly in 3D in the first place but It would also 
maka possible the re-processIng of the vast archives of 
5 2D film and video material for re-release to both the cin- 
ema and video markets. 

[0007] It would however be advantageous to be able 
to convert an existing 2D image so that it can be viewed 
as a 3D image. One way of achieving this is to convert 
^0 a single 2D image to two separate left and right images 
by a 'cut and paste' technique. In this technique, an ob- 
ject Is 'cut' from the Image and laterally displaced left or 
right then 'pasted' back onto the original image to pro- 
duce the required separate images. This however re- 
'5 suits In a blank region in the area fomially occupied by 
the object within the image. 

[0008] It is therefore an object of the present invention 
to overcome or minimise at least one of these problems. 
[0009] With this In mind, the present invention pro- 
vides in one aspect, a method of producing left and right 
eye images for a stereoscopic display from an original 
2D Image, wherein selected areas of said original image 
are displaced by a determined amount and direction to 
thereby generate stretched Images, said stretched Im- 
ages fomiing said left and right eye Images. 
[0010] The two converted images when respectively 
viewed by the left and right eye of a viewer can provide 
a 3D image without any blank region as would be the 
case with images produced by the 'cut and paste' tech- 
nique. 

[001 1 ] This document covers some of the major algo- 
rithmic processes involved in the conversion of 2D me- 
dia to 3D format, a new composite data format which is 
suitable for the communication and storage of this new 
3D media. It also discusses several hardware Imple- 
mentations for the real-time encoding, transmission and 
decoding of this new 3D format. 
[0012] The main benefits of the technology is that 
there are significant cost saving and media supply ad- 
vantages.ie. Only one camera has to be used forfllmlng. 
Due to the nature of the 2D to 3D conversion process it 
is possible to package and transfer the Image media vir- 
tually unchanged except for the addition of a small pack- 
et of 3D data that doesn't in any way encumber the final 
2D display process. In fact It enables the images to be 
displayed on standard 2D televisions in 2D or 3D (usage 
of shutter glasses or similar required) without degrada- 
tion of image quality whilst also allowing display on 3D 
televisions or other displays in 3D. 
[0013] The final stage of the 2D to 3D conversion 
process Is completed in real-time at the receiver and 
thus the increased bandwidth requirements for display- 
ing 3D images is local to the television decoder and does 
not adversely affect the channel handling capacity of the 
television carrier, 

[0014] According to another aspect of the present in- 
vention there is provided a method of describing the 
changes to be made to an original 2D image forconvert- 
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ing said 2D image Into stretched images for a stereo- 
scopic display. 

[001 5] According to a further aspect of the present in- 
vention, there is provided a method of encoding a video 
signal of a 2D image for allowing conversion of said vid- 
eo 2D image to stretched images for a stereoscopic dis- 
play. 

[001 6] According to yet another aspect of the present 
invention, there is provided a method of receiving a vid- 
eo signal of a 2D image including encoding data» and 
extracting the encoding data from the video signal for 
allowing conversion of the 2D image to stretched imag- 
es for a stereoscopic display. 

[0017] According to a further aspect of the present in- 
vention, there is provided a method of manipulating a 
2D video image with encoding data to thereby provide 
stretched images for a stereoscopic display. 
[0018] According to another aspect of the present in- 
vention, there is provided a method of producing left and 
right eye images for a stereoscopic display from an orig- 
inal 2D image including the steps of: 

a) identifying at least one object within said original 
image, 

b) outlining said or each object, 

c) defining a depth characteristic for said or each 
object, 

d) respectively displacing selected areas of said or 
each image by a determined amount in a lateral di- 
rection as a function of the depth characteristic of 
said or each object, to fonn two stretched images 
for viewing by the left and right eyes of the viewer. 

[0019] These image pairs may be either mirrored or 
similar to each other so that the stereoscopic 3D effect 

is optimised. 

[0020] The image may include a plurality of objects 
with each object being provided with a said respective 
depth characteristic. Images may be converted on an 
individual basis. Alternatively, a series of related images 
as in a video or film may be converted. 
[0021 ] The image may be digitised and the image may 
be stretched or converted electronically by temporarily 
placing a mesh over the image, the mesh Initially having 
a plurality of parallel lateral mesh lines and a plurality of 
parallel longitudinal mesh lines positioned at right an- 
gles to the lateral mesh lines. Each intersection of the 
mesh lines on the mesh may provide a mesh sub-point. 
The image can move together with the mesh so that dis- 
tortion of the mesh results in the stretching of the under- 
lying image. The mesh lines may remain continuous to 
provide for a smooth stretching of the image. The 
amount of displacement of each of the mesh sub-points 
from their initial position may provide the conversion da- 
ta for said original image. The sub-points may be dis- 
placed in a lateral direction. 

[0022] The displacement of the mesh sub-points may 
also be defined by a mathematical algorithm to thereby 
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provide for automatic conversion of Images. Further en- 
hancements to the method could be to add shadow, 
blurring and motion interpolation data to the conversion 
data including force paralex information and field delay 

5 and direction for motion paralex delays. 

[0023] It would be advantageous to be able to use ex- 
isting image transmission systems to transmit an image 
that can be viewed as a 3D Image. The present Invention 
can be applicable for use in image transmission s^tems 

10 sending video signals that provide 2D images. 

[0024] According to a further aspect of the present in- 
vention there is provided a method of producing a set of 
'object scripting' data from the conversion/stretching 
process that describes which objects in an Image are 

15 selected for processing, how they will be processed, 
their priority or otherwise over other objects and their 
depth characteristics. This scripting data may be stored 
in the memory of a computer for later usage to re-proc- 
ess the original 2D images or transmitted to another site 

20 (assuming the other site has the same 2D images) for 
reproduction of the 3D images. 
[0025] Therefore, according to a further aspect of the 
present Invention there is provided an encoder for en- 
coding a video signal providing a 2D video image, in- 

25 eluding: 

adding conversion data to the video signal to pro- 
vide the encoded signal, the data defining the displace- 
ment of respective selected points of the video image 
for converting the video image to a stretched image for 

30 a stereoscopic display. 

[0026] By adding the conversion data to the video sig- 
nal, existing transmission systems can be used to trans- 
mit the encoded signal. Various arrangements may be 
provided to add the conversion data to the video signal. 

35 For example, the data may be Included in the blank lines 
of the transmitted video image at the top and bottom of 
the image or in the horizontal sync period or horizontal 
overscan regions of each line. 
[0027] This invention Is not limited to the conversion 

40 of existing 2D video images. Rather the process can be 
readily used to produce the conversion data simultane- 
ously with the creation of the 2D video image. 
[0028] Therefore, according to a further aspect of the 
present invention there is provided a method of produc- 

45 Ing a 2D video image encoded with 3D conversion data 
including: 

capturing video images from a plurality of video 
cameras; 

50 comparing the video images from each respective 
video camera in order to produce the conversion 
data, the conversion data defining the displacement 
of respective points of the video image for convert- 
ing the video image to a stretched image for a ster- 

55 eoscopic display; and 

combining the video signal from one of said video 
cameras with the conversion data to thereby pro- 
' duce the encoded video signal. 
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[0029] in a further aspect of the present invention 
there is provided a method of producing a 2D video im< 
age encoded with 3D conversion data including: 

capturing left and right eye video Images from a s 
stereoscopic video camera; 
comparing the left and right eye video images from 
the stereoscopic video camera In order to produce 
the conversion data, the conversion data defining 
the displacement of respective points of the video 10 
image for converting the video image to a stretched 
image for a stereoscopic display; and 
combining the video signal from said video camera 
with the conversion data to thereby produce the en- 
coded video signal. is 

[0030] In yet a further aspect of the present invention 
there is provided a system of producing a 2D video sig- 
nal encoded with 3D conversion data including: 

20 

at least a first and second video camera displaced 
laterally relative to each other; 
conversion means for producing the conversion da- 
ta, said conversion means receiving data from said 
video cameras and comparing the data to thereby 25 
produce the conversion data, said conversion data 
defining the displacement of respective points of the 
video image from one of said video cameras for 
converting the video image to a stretched image for 
a stereoscopic display; 30 
an encoder means for combining the video signal 
trom said one video camera with the conversion da- 
ta from said conversion means to thereby produce 
the encoded video signal. 

35 

[0031] Where the 2D video image encoded with the 
3D conversion data is only required for a single viewer, 
only two video cameras are required, each camera rep- 
resenting the view seen by the left and right eyes of the 
viewer. ^ 
[0032] In yet another aspect the present Invention 
provides a system of producing a 2D video signal en- 
coded with 3D conversion data Including: 

a stereoscopic video camera; 45 
conversion means for producing the conversion da- 
ta, said conversion means receiving data from said 
video camera and comparing the data to thereby 
produce the conversion data, said conversion data 
defining the displacement of respective points of the so 
video image for converting the video Image to a 
stretched image for a stereoscopic display; 
an encoder means for combining the video signal 
with the conversion data from said conversion 
means to thereby produce the encoded video sig- ss 
nal. 

[0033] According to yet another aspect of the present 



6 

invention, there Is provided a decoder for decoding a 
video signal for providing a stereoscopic display, the sig- 
nal providing a 2D video Image and further including 
conversion data for converting the video image, the con- 
version data defining the displacement of respective 
points of the video image for converting the video image 
to a stretched image for a stereoscopic display, the de- 
coder including: 

a) means for receiving the video signal; and 

b) decoding means for reading the conversion data 
and for controlling the video signal to thereby pro- 
vide a converted video signal. 

[0034] The decoder may include 

a) an RGB or component video converter for con- 
verting the video signal into separate video compo- 
nents thereof, 

b) analog to digital conversion means for converting 
each video component to a respective digital signal, 
and 

c) digital storage means for storing said digital sig- 
nals. 

[0035] The decoding means may control a variable 
frequency clock means controlling the read out rate of 
the digital storage means whereby the storage means 
are read out at a variable rate. This results in the video 
image being stretched or compressed in accordance 
with the conversion data. 

[0036] Alternatively, the RGB or video components 
may be read into the storage means at a variable rate, 
and read out of the storage means at a fixed rate. 
[0037] The decoder may process a single video line 
or may also process multiple lines such as in a complete 
field or frame. In this case the full mesh from the con- 
version data is restored with pixel distortions ( lateral 
shifts) being calculated over the complete field or 
frame. 

[0038] The storage means may be in the form of a du- 
al port RAM line store. 

[0039] A digital to analog conversion means may be 
provided for converting the read out digital signal to a 
converted video signal to enable viewing on viewing 
means. The viewing means may include a television or 
other screen for viewing the converted video Image. The 
viewing means may further include shutter glasses con- 
trolled by the decoder to enable the converted video im- 
age to be viewed as a stereoscopic image. 
[0040] Alternatively, the decoder may include parallel 
storage means for storing digital signal for the converted 
lett and right video images respectively. The viewing 
means may then include a display unit for simultaneous- 
ly projecting the left and right video images. 
[0041] The decoder means may include separation 
means for separating the conversion data from the video 
signal. 
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[0042] According to a further aspect of the present in- 
vention, there Is provided a stereoscopic Image display 
system including: 

a) an encoder for encoding a video signal providing 
a video image with conversion data, the conversion 
data defining displacement of respective points of 
the video image for converting the video Image to a 
stretched image for a stereoscopic display; and 

b) a decoder for separating the conversion data 
from the video signal and for converting the video 
signal as a function of the conversion data. 

[0043] According to yet another aspect of the present 
invention, there is provided a multiviewer stereoscopic 
display system including: 

a) a decoder for decoding a video signal for provid- 
ing a stereoscopic display, the signal providing a 2D 
video Image and further including conversion data 
for converting the video image, the conversion data 
defining the displacement of respective points of the 
video image for converting the video image to a 
stretched image for a stereoscopic display, the de- 
coder Including means for receiving the video signal 
and decoding means for reading the conversion da- 
ta and for controlling the video signal to thereby pro- 
vide a converted video signal; 

[0044] It will be convenient to further describe the in- 
vention by reference to the accompanying drawings 
which illustrate possible implementations of the present 
Invention. 

[0045] Other implementations of the invention are al- 
so possible and consequent the particularities of the ac- 
companying drawings is not to be understood as super- 
seding the generality of the preceding description. 
[0046] In the drawings: 

Figure 1 shows an original image and conventional 
left and right images for providing a 3D or stereo- 
scopic Image; 

Figure 2 shows an original image and left and right 
images for providing a 3D image produced using a 
cut and paste technique; 

Figure 3 shows an original image and an image gen- 
erated by the Dynamic Depth Cueing (DDC) meth- 
od according to the present invention; 
Figure 4 shows a left and right image and the re- 
sultant 3D image according to the present inven- 
tion; 

Figure 5 shows an image distorted discontlnuously 
by a distortion mesh; 

Figure 6 shows an Image distorted continuously by 
a distortion mesh; 

Figure 7 shows example Mesh Spatial Displace- 
ment (MSD) data for a left and right mesh; 
Figure 8 shows a flow chart illustrating how MSD 
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data is added to a video Image according to the 
present invention; 

Figure 9 shows a block diagram indicating how a 
DDC decoder is incorporated in a video chain ac- 

5 cording to the present invention; 

Figure 10 shows a block diagram indicating a pos- 
sible implementation of a DDC decoder unit accord- 
ing to the present Invention providing field sequen- 
tial composite video outputs; 

10 Figure 1 1 shows a block diagram of another possi- 
ble Implementation of a DDC decoder unit accord- 
ing to the present invention providing field parallel 
composite video outputs; 

Figure 12 shows a block diagram of one version of 
'5 a MSD decoder according to the present invention; 
Figure 13 shows how MSD data is encoded in the 
composite video signal; 

Figure 1 4 shows a block diagram of an arrangement 
for providing real time generated DDC encoded vid- 
20 eo images; 

Figure 15 shows a block diagram of an alternative 
arrangement for providing real time generated DDC 
encoded video images; 

Figure 16 shows the principle of operation of multi- 
25 viewer 3D systems; 

Figure 17 shows a block diagram illustrating the 
principle of operation of a lenticular lens based 3D 

system; 

Figure 18 and 19 respectively show a multi-projec- 
30 tor system using a lenticular lens assembly; and 
Figure 20 shows a multi-projector system Incorpo- 
rating a DDC decoder according to the present In- 
vention. 

35 [0047] The method according to the present invention 

for enabling 2D or "monoscopic" video signals to be con- 
verted to 3D or "stereoscopic" video signals is referred 
to as Dynamic Depth Cueing (DDC) in the following de- 
scription and embraces the following but is not limited 
40 to these techniques: 

a) 3D GENERATION - A technique and procedure 
for converting 2D Images 

into 3D stereoscopic image pairs and for producing 
45 the 3D conversion data. 

b) 3D SCRIPTING - A technique to describe the 
changes required to be made to a 2D image in order 
to convert it to a 3D stereoscopic image pair. De- 
scribes which objects are selected, how they are 

50 processed and provides for a means of storage of 
3D data. 

c) 3D DATA ENCODING - A technique for adding 
information to a 2D video image in a defined format . 
The resulting modified video is compatible with ex- 

55 isting video recording, editing, transmission and re- 
ceiving systems. 

d) 3D STANDARDISED PROTOCOL - The 3D Con- 
version data is added to the 2D video using a de- 
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fined data format or standardised protocol. This pro- 
tocol may well become a world-wide standard for 
adding 3D Conversion data to 2D transmissions. 

e) 3D DATA DECODING - A technique for receiving 
the 2D video image plus cx)nversion data and to ex- 
tract the information added to a 20 video image so 
that a 3D stereoscopic Image pair may be synthe- 
sised. 

f) 3D SYNTHESIS - A technique to manipulate the 
2D video Image using the conversion data to syn- 
thesise a 3D stereoscopic image pair. 

[0048] In order to convert a 2D image to a simulated 
3D image it is necessary to modify the original image to 
produce two slightly different images and present these 
separate images to the left and right eyes independent- 
ly. 

The modification to the original image consists of a lat- 
eral shift of objects within the Image plane (located at 
the projection or viewing screen) in order to give the im- 
pression of depth. 

[0049] To make an object in an image appear farther 
away from the viewer, with respect to the image plane, 
then it is necessary to present the object within the im- 
age to the left eye with a slight left lateral shift and that 
to the right eye with a slight right lateral shift. This is 
illustrated in Figure 1 . To make an object appear to the 
viewer to be closer, it is necessary to shift the object 
within the image for the left eye laterally to the right and 
the object within the image for the right eye laterally to 
the left. For an object to be positioned at the image plane 
the object Is placed in the image at the same position 
for both eyes. 

[0050] When viewing objects in the real worid, a view- 
er also makes use of focus Information. However, with 
simulated 3D this information is not present and if the 
lateral shifts are made too great, particulariy in order to 
bring an object closer to the viewer, then the object ap- 
pears to break into two separate images and the 3D ef- 
fect Is tost. 

[0051] The left and right images can be produced us- 
ing a computer. The image is firstly digitised using a vid- 
eo digitiser and the resulting data stored in memory. The 
two new images can then be generated. 
[0052] The simplest way to generate the new left and 
right images with the required lateral shift is to "cut" the 
objects from the image and "paste" them back with the 
necessary lateral displacement, this will be referred to 
as the "cut and paste" technique. This can be done by 
first defining the position of an object to be "moved" by 
identifying it then "cutting" the object from the image and 
moving it laterally. 

[0053] The problem with this simple technique is that 
once the selected object is moved the background is al- 
so removed and a blank region in the background re- 
sults, see Figure 2. 

[0054] According to the present invention, an object 
within an Image Is "stretched" to provide the necessary 
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lateral shift and retain the original background detail. 
The resulting lateral distortions of the image are 
smoothed mathematically so that the resultant effect Is 
perceived as 'real' 3D with little or no visual artefacts. 

5 [0055] To better visualise the effect of this stretching 
on the image, imagine that the image to be convened 
has been printed onto a thin sheet of mbber. It is possi- 
ble to pick a point on the surface of the Image, adjacent 
to an object, and stretch It into a new position, to the 

10 right of its original position for example. The section of 
the image to the right of the object Is therefore com- 
pressed and that to the left stretched, see Figure 3. To 
the viewer the object now appears distorted if viewed by 
both eyes. 

15 [0056] However, if a similar but oppositely stretched 
Image is presented to the other eye, then the viewer 
does not see a distorted image, rather an object that has 
3D characteristics. This Is illustrated in Figure 4. 
[0057] The "stretching" of an object within an Image 

20 can be undertaken electronically. The objects of interest 
in each video frame are firstly identified by outlining 
them. For each object a depth or mesh distortion char- 
acteristic is also defined. The stretching can be conduct- 
ed by enabling an operator to stretch the image and view 

25 the effect of the resulting 3D image In real time. Operator 
skill and artistic Intervention can be used to determine 
the 3D Impact of the resulting Image and subsequent 
video sequence. Whilst individual video frames could be 
converted manually (ie. non real time) we also envisage 

30 automatically (ie. real time) converting a series of relat- 
ed frames that form a video "clip". The operator will de- 
fine the start and end frames of the video clip to be con- 
verted. They will also determine the relative depth of 
each object, relative to the image plane, in the start and 

35 end frames. The video clip will be processed using the 
start and end positions and depth of each object in the 
clip, to interpolate the required stretching or manipula- 
tion for the intermediate frames. 
[0058] In the case of multiple overiapping objects 

40 which have different depths, foreground objects are giv- 
en priority. This is Inherently the case, since the original 
2D image has been captured with a single camera, thus 
the pixel information has automatically been prioritised 
to the foreground. 

^ [0059] This "stretching" of the Image can be undertak- 
en electronically by manipulating the digitised image. A 
mesh (grid) is temporarily placed over the Image to be 
distorted such that, prior to distortion, the co-ordinates 
of each row and column of the mesh is 0,0, The mesh 

so X co-ordinates are altered which results In the undertying 
image being distorted. Rather than just the image area 
immediately under the mesh line being moved, which 
would result in a discontinuity - figure 5, adjacent mesh 
lines are also moved to produce a smooth distortion, 

55 Figure 6. 

[0060] The coarseness of the distortion mesh deter- 
mines the impact of the 3D effect. The coarser the mesh 
the more splined other objects closer to the object being 
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stretched are impacted. This results in a lower 3D im- 
pact in the resulting image. A finer mesh results in sharp- 
er edges to the objects, a higher impact 3D effect, but 
greater edge discontinuities. The order of the distortion 
mesh will for explanation purposes be assumed to be 5 
16X16. Information on each subpoint on the mesh (le. 
co-ordinate positions alter distorting) are encoded so as 
to produce background and foreground subpoints. For 
example, 4 bits can be used for the subpoint encoding 
which will result in 16 different levels, 4 background and 
12 foreground. The fomnat of the subpoint encoding can 
also be determined by experimentation and adjusted to 
suit the application. 

[0061 ] Alternatively, this mesh distortion process may 
be defined by a mathematical algorithm which would en- 
able automatic processing of Images. 
[0062] Note that once the mesh distortions for the left 
eye have been determined, then the co-ordinates of the 
distortions for the right eye are simply obtained by scalar 
multiplication of the matrix by -1 ( ie. shifted In the op- 
posite lateral direction by the same amount) and can be 
calculated automatically. This is illustrated in Figure 7. 
[0063] The matrix that is formed from the relative hor- 
izontal offset of each intersection point of the distorted 
mesh defines the Mesh Spatial Displacement (MSD) da- 
ta. 

[0064] In order to fully define and reproduce the re- 
sultant 3D image all that is necessary is to provide the 
original, unaltered, 2D image and the MSD data. Thus 
3D images may be stored, transmitted, generated, ed- 
ited, and manipulated by considering the 2D image and 
an associated MSD data file. 

[0065] It is therefore possible to store and transmit 3D 
images over conventional 2D video systems by encod- 
ing MSD data within each video frame. Since the original 
2D video image is stored and can be transmitted without 
alteration, the resulting video is fully compatible with all 
existing video and television systems. Existing 2D TV 
receivers will display a nonmal picture. 
[0066] A number of existing techniques can be used 
to add the MSD data to the 2D image such that it is not 
detected by the viewer and is compatible with existing 
video standards. These techniques include, but are not 
limited to: 

a) inserting the MSD information in the spare lines 
at the top and bottom of the picture that are set at 
black level, ie. in a similar manner to the addition of 
"Teletext" data; 

b) in the unseen over-scan region at the left and 
right of each image; 

c) In the horizontal sync period, along the lines of 
the British Broadcasting Corporation "sound in 
sync" system. 

[0067] In the future, with the introduction of digital 
HDTV, spare digital data frames will be available to in- 
sert the MSD data. 
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[0068] The process of adding the MSD data to a 2D 
video image to form a DDC encoded video frame is il- 
lustrated in Figure 8. 

[0069] The amount of MSD data is small, estimated 
to be approximately 100 bytes per frame. This can be 
further compressed if necessary, for storage and trans- 
mission, by using standard data compression tech- 
niques such as run-length or differential encoding. 
[0070] Because of the small amount of data, the re- 
quired data rate Is also low. It Is also possible to use 
spatial and temporal compression to further reduce the 
data required since the MSD data does not vary rapidly 
over a number of frames. The exact time relationship 
between the MSD data and its associated frame is not 
critical, a displacement error of one frame is probably 
acceptable. 

[0071 ] Again due to the small amount of data, low data 
rate and non-critical alignment, the MSD data could be 
sent over a number of frames ie. four frames with a quar- 
ter of the information in each frame. 
[0072] A block diagram indicating how a DDC decod- 
er is incorporated in a video chain is shown in Figure 9. 
DDC encoded video, which can be obtained via any ex- 
isting video source I.e. terrestrial, satellite etc, is applied 
to the input of a DDC decoder. One output of the decoder 
is a standard video wavefomn (or video modulated radio 
frequency signal) that will drive a standard TV display 
and enable a viewer wearing shutter glasses, synchro- 
nised by the DDC decoder, to view a 3D image. 
[0073] Additionally outputs are available from the 
DDC decoder to drive other 3D display devices such as 
Virtual Reality headsets or an autostereoscopic display 
as described in Australian application No. 66718/94. 
[0074] One possible Implementation of the DDC de- 
coder is shown in Figure 10. Incoming video, which can 
be PAL or NTSC, In composite or S-video format, is ap- 
plied to a composite RGB or component video convert- 
er. Each of the RGB or component video outputs is ap- 
plied to an analog to digital converter and the digital out- 
put fed to the input port of a dual port RAM line store. 
Each tine of digital video data is entered into RAM at a 
constant speed. The data is read out of the line store at 
a rate determined by a variable frequency clock control- 
led by the output of the MSD decoder. 
[0075] The effect of reading out the line data from the 
RAM at a variable rate causes the resulting video to be 
stretched or compressed in accordance with the MSD 
data. 

[0076] The converted data Is then applied to a digital 
to analog converter and a PALTNTSC encoder. The re- 
sulting 3D field sequential composite video signal is ap- 
plied to the display. (Note: this process can also operate 
with the video signals read into the line stores at a var- 
iable rate and read out at a fixed rate. It is necessary to 
convert the incoming composite video signal to RGB or 
component video since, as the output from the line store 
is read out at a variable rate, the chrominance frequency 
would change and cause display errors.) 
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[0077] DDC decoding may also be imptemented us- 
ing a field or frame store. In this case the full mesh from 
the MSD data is restored with pixel distortions (lateral 
shifts) being calculated over the complete field or frame. 
[0078] The 3D stereoscopic Image pair is then dis- 
played from the resulting RGB or component video out- 
put. 

[0079] The shutter glasses controller provides an In- 
fra-red tight source that provides timing pulses for the 
shutter glasses. The controller is synchronised by the 
PAITNTSC encoder. Additionally, the controller com- 
mands the shutter glasses to remain open during 
scenes that are not 3D encoded, or not suitable for 3D 
encoding, thus providing improved image quality during 
those portions of the video sequence. 
[0080] Figure 11 shows the block diagram of a DDC 
decoder that produces 3D field parallel composite video 
outputs. The MSD decoder produces two variable 
speed clocks, one for the left and one for the right RAM 
line stores. This form of decoder is suitable for replacing 
the field stores in the applicants' existing autostereo- 
scopic 3D displays. Note that this technique provides 
left and right video sources that are at the same field 
rate as the original 2D video source, ie field sequential 
video output is produced. 

[0081] Alternatively a non-field video output may be 
produced at the output at a higher resolution scanning 
rate than the 2D image. 

[0082] Figure 12 shows a block diagram of one ver- 
sion of a MSD decoder. In this case it is assumed that 
the MSD data is encoded in the composite video signal 
in the first 16 lines of the video signal, Figure 13. Incom- 
ing composite video is fed to a sync separator which pro- 
vides vertical and horizontal timing signals for the mi- 
croprocessor. Additional the video is fed to a black level 
clamp circuit, comparator and level shifter The output 
from the level shifter is a TTL level signal which contains 
the serial encoded MSD data on lines 1 to 1 6 of the video 
signal. The microprocessor loops waiting for a horizon- 
tal sync pulse, after determining the horizontal sync 
pulse for line 1 the microprocessor reads the next 16 
bytes. A similar process is repeated on the following 15 
lines until the MSD data has tDeen read, Based upon the 
MSD data received the microprocessor provides the 
variable speed clock for the digital video line store on 
each subsequent video line. The microprocessor main- 
tains an index to which video line is being processed by 
counting the line sync pulses. 
[0083] Depth perception of 3D images varies between 
viewers. Also when viewing 3D Images with shutter 
glasses the "strength" of the 3D Image requires adjust- 
ment with viewing distance. The strength of the 3D im- 
age can be altered with a remote control device which, 
via an algorithm applied by the microprocessor, allows 
the strength of the 3D image to be varied by the viewer. 
The algorithm alters the magnitude of each element in 
the MSD matrix thus altering the strength of the 3D ef- 
fect. It is anticipated that once the preference for a par- 



ticular viewer has been entered then this setting will be 
maintained by the decoder unit. 
[0084] There are many techniques for the real time 
generation of DDC encoded video images. In one such 
5 technique the distance between the camera lens and an 
additional, range finding, Charge Coupled Device 
(CCD) array is altered - Figure 14. This will produce a 
series of frames with each object in the image at varying 
stages of focus. A sharpness detecting algorithm is then 
10 run across the series of frames and a sharpness index 
for each object in the image detemriined. We then deter- 
mine on which frame each object was the sharpest, this 
indicates on which focal plane the object is on. This in- 
formation is then used to form the MSD data. 
15 [0085] Figure 15 indicates another technique where- 
by two video cameras are used to create separate left 
and right eye Images. The luminance infomfiation from 
each camera is digitised and fed to a line store. An au- 
tocorrelator, or similar operation, compares the bit pat- 
terns within the two line stores (left and right by 16 ele- 
ments) and looks for a match. The difference (distance) 
between the video patterns, which represents objects in 
the image, is used to produce the MSD data. One of the 
camera outputs is combined with the MSD data in real 
time to produce DDC encoded video. 
[0086] Alternatively a stereoscopic video camera 
could be used in place of the two video cameras. 
[0087] DDC may be used to overcome a serious 
shortcoming of existing, non eye-tracking, autostereo- 
scopic 3D multiviewer systems. These systems provide 
a 3D image by producing a repeating sequence of left- 
right images as Illustrated in Figure 1 6. The distance be- 
tween each successive image is 65 mm which is equal 
to the distance between the viewers eyes. Therefore a 
viewer located at position A will see a correctly se- 
quenced 3D image. 

[0088] However, should the viewer move laterally 32 
mm, or be at position B, then the left image will be seen 
by the right eye and the right image by the left eye. ie. 
the viewer sees a "reverse 3D" image. The reverse 3D 
image is very uncomfortable to view and after a short 
while causes viewer distress ie. headache. 
Most multiviewer autostereoscopic system have this 
shortcoming. In particular those based upon lenticular 
lenses and grid type image separators. A multiviewer, 
lenticular lens based, autostereoscopic system is illus- 
trated in Figure 17. The image from the left projector 
passes through the first lenticular lens and is focused 
on the surface of a matt screen. A second lenticular lens 
refocusses this image to form a vertical stripe of light at 
the viewing distance. A second projector, containing the 
right image, also illuminates the first lenticular lens, how- 
ever, due to a lateral displacement between the two pro- 
jectors, the resulting right eye image, at the viewer 
plane, is displaced 65 mm from the left image. This se- 
quence of altemate left right images, spaced 65 mm 
apart, is repealed. 

[0089] A viewer in the correct position views a correct 
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30 image, however, reverse 3D images are also pre- 
sented should the viewer move or be incorrectly posi- 
tioned as explained above. 

[0090] In practice, when first sitting down to view a 
lenticular lens based 3D system it is difficult for the view- 
er to determine if a correct or reverse 3D image is being 
viewed, tt is not until discomfort is felt that the viewer Is 
aware of being incorrectly positioned. 
[0091] Additionally, it is difficult for a viewer to main- 
tain the correct viewing position for an extended period 
of time. It is also necessary that the viewer be positioned 
the correct distance from the second lenticular lens, 
since if viewed at the incorrect distance morae fringing 
and cross talk effects tend to occur. 
[0092] An additional problem with lenticular lens 
based system is resolution. The resolution Is limited by 
the pitch of each *lens-let" in the overall lenticular lens, 
which is in the order of 1 mm across. 
[0093] Instead of projecting just left and right images, 
consider a lenticular lens based system using a number 
of projectors, say four, such as to produce a series of 
images 1 ,2.3,4 each spaced 65 mm apart, as per Figure 
18. The original scene has been recorded using four 
cameras with the same sequence and spacing. A viewer 
at positions A, B or D sees a correct 3D image whilst at 
position C the viewer sees a reverse 3D Image. 
[0094] This is a substantial improvement over the pre- 
vious left right system since now an acceptable 3D im- 
age Is viewable over 3 times the lateral distance. Note 
that, as with the previous left right system, a viewer will 
not be aware, until discomfort is felt, that position D is 
providing a reverse 3D Image. 
[0095] If now projector four Is replaced with a "null" 
image (black), Figure 19, then positions A and B wori< 
as before, position C produces a monoscopic image In 
the right eye and a black image in the left, which is still 
viewable without viewer discomfort. Similariy position D 
produces a monoscopic image but again without a re- 
verse 3D effect. We thus have a system where 50% of 
the viewer locations produce a correct 3D image and 
50% a slightly inferior, although not objectionable, im- 
ages and the reverse 3D effect has been eliminated. 
[0096] By increasing the number of projectors, and in- 
cluding a null image, the lateral distance over which a 
correct 3D image can be viewed is increased and the 
reverse 3D effect has been eliminated. 
[0097] It is however not considered practical to imple- 
ment such a multi-projector system, primarily because 
the transmission/recording bandwidth necessary to pro- 
vide video images to each projector becomes impracti- 
cal as the number of projectors is increased. 
[0098] The limitations of this approach can be over- 
come by transmitting a DDC encoded 2D image and us- 
ing DDC to synthesise the required number of projection 
Images. Whilst a wide bandwidth is required within the 
DDC decoder, the original 2D bandwidth is maintained 
at the transmitter and/or recording equipment. 
[0099] A multiviewer 3D, lenticular lens based, sys- 
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tern using a DDC decoder is shown in Figure 20. 
[0100] DDC encoding/decoding will enable the pro- 
duction of a sequence of video images that represent 

the range of possible images from the extreme left of 
5 the original image to the extreme right, as follows:- 

[L, L«|, L2 Rv R] 

[0101] In summary some of the applications may be 
described by: 

[0102] DDC, or [Dynamic Depth Cueing, is a term for 

10 a form of data derived from the 2D to 3D conversion 
process which is in an intemnediary stage of conversion. 
At this stage the data consists of the original video signal 
and a data packet (encoded either in digital or analogue 
form) such that this additional data is all that is required 

15 to instaict specific electronics hardware and/or software 
to complete the conversion task. The resultant 3D infor- 
mation can take the form of either a field sequential (ie. 
left/right) type video format, two separate video streams, 
a line be line system (Ie. 1 line from left field, 1 line form 

20 right field), or some other fonnat that is advantageous. 
[0103] Through careful design of the format of the 
conversion data packet it Is possible to include this ad- 
ditional data such that it is not noticed when displayed 
on a standard TV. Thus it becomes possible to introduce 

25 3D television transmissions without upsetting the exist- 
ing television infra-structure. The decoder could be lo- 
cated near to the viewing device (le.TV) and essentially 
be a 'black box' that intercepts the transmission signal 
decodes it and then outputs it to the TV for viewing. 

30 Therefore upgrading of existing 2D pay TV or television 
network structures would be simplified by the addition 
of a 'black box' for each TV. 

[0104] In the provision of media for MULTI-IMAGE 
type auto-stereoscopIc 3D displays systems. These 

35 systems rely on the provision of multiple Images with 
slightly differing perspective views. They can be quite 
effective if the number of different views is large (Say 
8-16) in that they allow true multiviewer capability. Their 
major drawback is that because they require many dif- 

^ ferent views to all be available at the same time the pro- 
vision of the media is extremely difficult even utilising 
sophisticated video compression techniques. However, 
if a DDC decoder is used to generate the 3D media it 
becomes possible to generate as many individual per- 

45 spective views as are required, and as this image equip- 
ment ie. TV's and video recorders, as normal 2D imag- 
es. The viewer is not aware of any changes to the trans- 
mitted image. 

[0105] Standard 2D video images which have been 
50 DDC encoded have the following characteristics: 

DDC encoded 2D images can be received on stand- 
ard video equipment Ie. TVs and video recorders, 
as normal 2D images. The viewer Is not aware of 
55 any changes to the transmitted image. 

DDC encoded Images are fully compatible with all 
existing video, editing, recording, receiving and 
transmission systems and technologies. Thus DDC 
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encoded 2D video Images are transparent to all ex- 
isting analog video and television technologies. 
DDC encoded 2D video can be introduced to the 
market place in a similar manner to the introduction 
of colour TV and stereo sound. Only TV sets fitted 
with DDC decoders (with viewers using appropriate 
viewing glasses), or 3D TVs, will display the 3D im- 
ages. 

DDC encoding enables seamless transition be- 
tween scenes that will benefit from 3D encoding and 
those that are more beneficially displayed in 2D. 
This transition is not noticed by the viewer. 
DDC encoded video may be displayed on all exist- 
ing 3D displays and is suitable for multi-viewer sys- 
tems, 

DDC encoding retains the original video source's 
line and field standards 

DDC encoding does not reduce the image update 
frequency as is the case when encoding 3D video 
images in Left/Right field sequential format. 



Claims 

1. A method of encoding a 2D video signal with con- 
version data to assist in the conversion of said 2D 
video signal to left and right eye Images for a ster- 
eoscopic display, characterised In that said meth- 
od includes: 

adding conversion data to said video signal to 
provide the encoded signal, the data defining 
the displacement of respective selected points 
of said video image for converting the video im- 
age to a stretched image for a stereoscopic dis- 
play. 

2. A method as claimed in claim 1 characterised in 
that said conversion data is transmitted in the blank 
tines at the top and/or bottom of a standard 2D im- 
age to be transmitted. 



10 



6. 



7. 



ed viewpoints in order to produce said conver- 
sion data; and said conversion data; and 
combining the video signal of the video images 
from one of said viewpoints with the conversion 
data to thereby produce the encoded video sig- 
nal. 

A method according to claim 5 wherein said video 
signals are produced from two separate cameras. 

A method according to claim 5 wherein said video 
signals are left and right eye video images from a 
stereoscopic video camera. 
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8. A method of decoding a video signal for providing 
a stereoscopic display characterised in that said 
method includes: 

receiving said video signal, said video signal in- 
cluding a 2D video image and conversion data 
for converting the video image, the conversion 
data defining the displacement of respective 
points of the video image for converting the vid- 
eo image to a stretched image for a stereoscop- 
ic display; 

reading the conversion data from said video 
signal; and 

producing left and right eye images from the 2D 
video image for display by displacing objects 
within said 2D video image in accordance with 
said conversion data. 

9. A method as claimed in claim 8 characterised in 
that said method further includes the steps of: 

converting said 2D video image into RGB com- 
ponents; 

converting each component into a digital signal; 
and 

storing said digital signals prior to production of 
the left and right eye images. 



3. A method as claimed in claim 1 characterised In 
that said conversion data is transmitted in the hor- 
izontal sync period of a standard 2D image to be 
transmitted. 



1 0. A method as claimed in claims 8 or claim 9 charac- 
terised In that said digital signals are read out of 
^ storage at a variable rate as a function of the con- 
version data. 



4. A method as claimed in claim 1 characterised In 
that said conversion data is transmitted in the hor- 
izontal overscan regions of each line of a standard 
2D image to be transmitted. 



11. A method as claimed in claim 8 or claim 9 charac- 
terised In that said digital signals are read into stor- 
50 age at a variable rate as a function of the conversion 
data. 



5. A method according to any preceding claim char- 
acterised In that said method includes: 

producing video signals of video images from 

two separate viewpoints; 

comparing the video images from the separat- 



12. A method as claimed in any one of claims 8 to 11 
characterised in that said digital signals are con- 

55 verted to analog for viewing on an analog system. 

13. A method as claimed in any one of claims 8 to 12 
characterised in that said conversion data is sep- 
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arated from said video signal. 

14. A system of producing a 2D video signal encoded 
with 3D conversion data characterised In that said 
system inciudes: 

means for producing video signals of video im- 
ages at two separate viewpoints; 
conversion means for producing the conver- 
sion data, said conversion means receiving the 
video signals from said means and comparing 
the signals from the separate viewpoints to 
thereby produce the conversion data, said con- 
version data defining the displacement of re- 
spective points of one of said video images for 
converting the video image to a stretched im- 
age for a stereoscopic display; 
an encoder means for combining the video sig- 
nal from said one video image with the conver- 
sion data from said conversion means to there- 
by produce the encoded video signal. 

15. A system according to claim 14 wherein said means 
comprises first and second video cameras dis- 
placed laterally relative to each other. 

1 6. A system according to claim 1 4 wherein said means 
comprises a stereoscopic camera for capturing left 
and right eye video images. 

17. A decoder for decoding a video signal for providing 
a stereoscopic display, the signal providing a 2D 
video image and further including conversion data 
for converting the video image, the conversion data 
defining the displacement of respective points of the 
video image for converting the video Image to a 
stretched image for a stereoscopic display, charac- 
terised in that said decoder includes: 

a) means for receiving the video signal; and 

b) decoding means for reading the conversion 
data and for controlling the video signal to 
thereby provide a converted video signal. 

18. A decoder as claimed in claim 17, characterised 
in that said decoder including: 

a) an RGB or component video converterfor 
converting the video signal into separate video 
components thereof; 

b) analog to digital conversion means for con- 
verting each video component to a respective 
digital signal; and 

c) digital storage means for storing said digital 
signals. 

1 9. A decoder as claimed in claim 1 7 or claim 1 8, char- 
acterised in that said decoder is adapted to control 



a variable frequency clock means controlling the 
read out rate of the digital storage means whereby 
the storage means are read out at a variable rate, 
to thereby convert the 2D image in accordance with 
5 the conversion data. 

20. A decoder as claimed In claim 1 7 or claim 1 8, char- 
acterised In that said decoder is adapted to control 
a variable frequency clocl< means controlling the 

10 read in rate of the digital storage means whereby 
the RGB or video components may be read into the 
storage means at a variable rate, to thereby convert 
the 2D image in accordance with the conversion da- 
ta. 

IS 

21 . A decoder as claimed in any one of claims 1 7 to 20 
characterised In that the storage means is in the 
form of a dual port RAM line store. 

20 22. A decoder as claimed in any one of claims 1 7 to 21 
characterised In that the decoder processes a sin- 
gle video line. 

23. A decoder as claimed in any one of claims 1 7 to 21 
25 characterised In that the decoder processes mul- 
tiple video lines. 

24. A decoder as claimed in any one of claims 17 to 23 
characterised in that a digital to analog conversion 

30 means is provided for converting the read out digital 
signal to a converted video signal to enable viewing 
on a viewing means. 

25. A decoder as claimed in any one of claims 17 to 23 
35 characterised In that said decoder includes a par- 
allel storage means for storing digital signal forthe 
converted left and right video images respectively. 

26. A decoder as claimed in any one of claims 17 to 23 
40 characterised in that said decoder includes a sep- 
aration means for separating the conversion data 
from the video signal. 

27. A stereoscopic image display system character- 
45 ised in that said system includes: 

a) an encoder for encoding a video signal pro- 
viding a video Image with conversion data, the 
conversion data defining displacement of re- 

50 spective points of the video image for convert- 

ing the video image to a stretched image for a 
stereoscopic display; and 

b) a decoder for separating the conversion data 
from the video signal and for converting the vid- 

55 eo signal as a function of the conversion data. 

28. A multiviewer stereoscopic display system charac- 
terised in that said system includes a decoder ac- 
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